Cooling electronic devices in a data center

ABSTRACT

A data center cooling system includes an outer container that defines a first volume; an inner container that defines a second volume and is positioned within the first volume, the inner container including an air outlet that includes an airflow path between the first and second volumes; a liquid seal to fluidly isolate a liquid phase of a non-conductive coolant that fills at least a portion of the first and second volumes from an ambient environment; and a plurality of electronic heat-generating devices at least partially immersed in the liquid phase of the non-conductive coolant to transfer a heat load to the non-conductive coolant.

TECHNICAL FIELD

This document relates to systems and methods for providing cooling toelectronic equipment, such as computer server assemblies and relatedequipment in computer data centers, with a non-conductive coolant.

BACKGROUND

Computer users often focus on the speed of computer microprocessors(e.g., megahertz and gigahertz). Many forget that this speed often comeswith a cost—higher power consumption. This power consumption alsogenerates heat. That is because, by simple laws of physics, all thepower has to go somewhere, and that somewhere is, in the end, conversioninto heat. A pair of microprocessors mounted on a single motherboard candraw hundreds of watts or more of power. Multiply that figure by severalthousand (or tens of thousands) to account for the many computers in alarge data center, and one can readily appreciate the amount of heatthat can be generated. The effects of power consumed by the criticalload in the data center are often compounded when one incorporates allof the ancillary equipment required to support the critical load.

Many techniques may be used to cool electronic devices (e.g.,processors, memories, networking devices, and other heat-generatingdevices) that are located on a server or network rack tray. Forinstance, forced convection may be created by providing a coolingairflow over the devices. Fans located near the devices, fans located incomputer server rooms, and/or fans located in ductwork in fluidcommunication with the air surrounding the electronic devices, may forcethe cooling airflow over the tray containing the devices. In someinstances, one or more components or devices on a server tray may belocated in a difficult-to-cool area of the tray; for example, an areawhere forced convection is not particularly effective or not available.

The consequence of inadequate and/or insufficient cooling may be thefailure of one or more electronic devices on the tray due to atemperature of the device exceeding a maximum rated temperature. Whilecertain redundancies may be built into a computer data center, a serverrack, and even individual trays, the failure of devices due tooverheating can come at a great cost in terms of speed, efficiency, andexpense.

SUMMARY

This disclosure describes a data center cooling system that cools one ormore electronic heat-generating devices with a non-conductive coolant.For example, the non-conductive coolant may be a dielectric coolant. Insome aspects, a non-conductive coolant or dielectric coolant does notconduct an electric charge or conducts a small enough electric chargethat operation of any electronic device immersed in such coolant is notaffected. Examples of a non-conductive coolant or a dielectric coolantinclude, for instance, aromatics, silicate-ester, aliphatics, silicones,fluorocarbons, and oils such as mineral oil. In some aspects, the datacenter cooling system includes one or more inner containers that fluidlyisolate the non-conductive coolant from an ambient environment alongwith an outer container that encloses the inner container. In someaspects, the data center cooling system includes a container for theelectronic devices that includes a chimney or vertically extendinghousing that, for example, can at least partially enclose a coolingmodule to cool the non-conductive coolant. In some aspects, theelectronic devices may be mounted with or to a server assembly thatincludes a filler to reduce a free (e.g., air filled) volume within oneor more containers of the data center cooling system.

In an example implementation, a data center cooling system includes anouter container that defines a first volume; an inner container thatdefines a second volume and is positioned within the first volume, theinner container including an air outlet that includes an airflow pathbetween the first and second volumes; a liquid seal to fluidly isolate aliquid phase of a non-conductive coolant that fills at least a portionof the first and second volumes from an ambient environment; and aplurality of electronic heat-generating devices at least partiallyimmersed in the liquid phase of the non-conductive coolant to transfer aheat load to the non-conductive coolant.

In a first aspect combinable with the example implementation, the outercontainer includes a pressure relief valve configured to vent a portionof air, vented from the second volume through the air outlet and intothe first volume, to the ambient environment.

In a second aspect combinable with any of the previous aspects, theouter container includes a base and a chimney that extends verticallyfrom the base, the base including a first portion of the first volumeand the chimney includes a second portion of the first volume that is influid communication with the first portion.

In a third aspect combinable with any of the previous aspects, thesecond portion defines a coolant recovery layer that includes a mixtureof air and a vapor phase of the non-conductive coolant, a vaporcondensing layer that includes substantially the vapor phase of thenon-conductive coolant, and a liquid sub-cooling layer that includessubstantially the liquid phase of the non-conductive coolant.

A fourth aspect combinable with any of the previous aspects furtherincludes a cooling module mounted in the chimney and configured to coolthe non-conductive coolant.

In a fifth aspect combinable with any of the previous aspects, thecooling module includes a cooling coil that includes a cooling fluidinlet and a cooling fluid outlet.

A sixth aspect combinable with any of the previous aspects furtherincludes a pump positioned in the liquid sub-cooling layer of the secondportion of the first volume.

A seventh aspect combinable with any of the previous aspects furtherincludes one or more nozzles that are fluidly coupled to the pump andpositioned in the vapor condensing layer of the second portion of thefirst volume.

In an eighth aspect combinable with any of the previous aspects, the oneor more nozzles is fluidly coupled to the pump through one or moreconduits that extend from the first volume into the second volume.

In a ninth aspect combinable with any of the previous aspects, thenon-conductive coolant includes a single-phase non-conductive coolant.

A tenth aspect combinable with any of the previous aspects furtherincludes a pump positioned in a liquid sub-cooling layer of the firstvolume.

An eleventh aspect combinable with any of the previous aspects furtherincludes a cooling module positioned in the first volume that extendsbetween the liquid sub-cooling level and a vapor condensing level of thefirst volume.

A twelfth aspect combinable with any of the previous aspects furtherincludes one or more nozzles that are fluidly coupled to the pump andpositioned in the vapor condensing layer of the first volume.

In a thirteenth aspect combinable with any of the previous aspects, theouter container includes a human-occupiable housing, and the firstvolume includes a human-occupiable workspace.

In a fourteenth aspect combinable with any of the previous aspects, thenon-conductive coolant includes a dielectric coolant.

In another example implementation, a method for cooling electronicheat-generating devices in a data center includes enclosing a pluralityof electronic heat-generating devices in a volume defined by a firstcontainer; immersing the plurality of electronic heat-generating devicesin a liquid phase of a non-conductive coolant; enclosing the firstcontainer in a second volume of a second container, the non-conductivecoolant filling at least a portion of the first and second volumes;sealing the liquid phase of the non-conductive coolant from an ambientenvironment; and transferring a heat load from the plurality ofelectronic heat-generating devices to the liquid phase of thenon-conductive coolant.

A first aspect combinable with the example implementation furtherincludes forming an airflow path between the first and second volumes.

A second aspect combinable with any of the previous aspects furtherincludes venting a portion of air from the first volume, through theairflow path, through the second volume and to the ambient environment.

In a third aspect combinable with any of the previous aspects, thesecond container includes a base and a chimney that extends verticallyfrom the base, the base including a first portion of the second volumeand the chimney includes a second portion of the second volume that isin fluid communication with the first portion.

A fourth aspect combinable with any of the previous aspects furtherincludes cooling a mix of air and a first portion of a vapor phase ofthe non-conductive coolant in a top portion of the chimney to condensethe first portion of the vapor phase to the liquid phase of thenon-conductive coolant.

A fifth aspect combinable with any of the previous aspects furtherincludes cooling a second portion of the vapor phase of thenon-conductive coolant in a middle portion of the chimney to condensethe second portion of the vapor phase to the liquid phase of thenon-conductive coolant.

A sixth aspect combinable with any of the previous aspects furtherincludes sub-cooling the liquid phase of the non-conductive coolant in abottom portion of the chimney.

A seventh aspect combinable with any of the previous aspects furtherincludes circulating a portion of the sub-cooled liquid phase of thenon-conductive coolant from the bottom portion of the chimney to thefirst volume to contact the plurality of electronic heat-generatingdevices.

An eighth aspect combinable with any of the previous aspects furtherincludes supplying a cooling fluid to a cooling module positioned in thechimney.

In a ninth aspect combinable with any of the previous aspects, thenon-conductive coolant includes a single-phase non-conductive coolant.

A tenth aspect combinable with any of the previous aspects furtherincludes circulating a sub-cooled liquid phase of the non-conductiveliquid coolant from a bottom portion of the second volume to a topportion of the second volume.

An eleventh aspect combinable with any of the previous aspects furtherincludes circulating the sub-cooled liquid in the top portion over acooling module positioned in the second volume.

In a twelfth aspect combinable with any of the previous aspects, thesecond container includes a human-occupiable housing, and the secondvolume includes a human-occupiable workspace.

In a thirteenth aspect combinable with any of the previous aspects, thenon-conductive coolant includes a dielectric coolant.

In another example implementation, a system includes a first housingadapted to enclose a plurality of server assemblies, at least one of theserver sub-assemblies including a top that sealingly engages the firsthousing and a server board coupled to the top; a second housing thatencloses the first housing; and a dielectric coolant enclosed within thefirst and second housings to cool the server board.

In a first aspect combinable with the example implementation, thedielectric coolant includes a liquid phase substantially containedwithin the second housing and a vapor phase substantially containedwithin the first housing.

A second aspect combinable with any of the previous aspects furtherincludes a cooling coil mounted in the second volume to cool the liquidphase of the dielectric coolant, and condense the vapor phase of thedielectric coolant to the liquid phase of the dielectric coolant.

In a third aspect combinable with any of the previous aspects, thesecond housing includes a base housing and an extension housing that iscoupled to the base housing and extends vertically from the basehousing.

In a fourth aspect combinable with any of the previous aspects, thecooling coil is positioned in the extension housing.

A fifth aspect combinable with any of the previous aspects furtherincludes a plurality of computing devices mounted on the server board.

A sixth aspect combinable with any of the previous aspects furtherincludes an I/O patch panel mounted on the top and communicably coupledto the plurality of computing devices mounted on the server board.

A seventh aspect combinable with any of the previous aspects furtherincludes a pump positioned in the liquid phase of the dielectric coolantto circulate the liquid phase of the dielectric coolant to a vapor phaselayer of the first housing.

In another example implementation, a data center cooling system includesa container that defines a volume, the container including a base and achimney that extends vertically from the base, the base including afirst portion of the volume and the chimney includes a second portion ofthe volume that is in fluid communication with the first portion; aliquid seal to fluidly isolate a liquid phase of a non-conductivecoolant that fills at least a portion of the volume from an ambientenvironment; and a plurality of electronic heat-generating devices atleast partially immersed in the liquid phase of the non-conductivecoolant to transfer a heat load to the non-conductive coolant.

In a first aspect combinable with the example implementation, thecontainer includes a pressure relief valve configured to vent a portionof air, vented from the second volume through the air outlet and intothe first volume, to the ambient environment.

In a second aspect combinable with any of the previous aspects, thesecond portion defines a coolant recovery layer that includes a mixtureof air and a vapor phase of the non-conductive coolant, a vaporcondensing layer that includes substantially the vapor phase of thenon-conductive coolant, and a liquid sub-cooling layer that includessubstantially the liquid phase of the non-conductive coolant.

A third aspect combinable with any of the previous aspects furtherincludes a cooling module mounted in the chimney and configured to coolthe non-conductive coolant.

In a fourth aspect combinable with any of the previous aspects, thenon-conductive coolant includes a single-phase non-conductive coolant.

A fifth aspect combinable with any of the previous aspects furtherincludes a pump positioned in a liquid sub-cooling layer of the volume.

A sixth aspect combinable with any of the previous aspects furtherincludes a cooling module positioned in the volume that extends betweenthe liquid sub-cooling level and a vapor condensing level of the volume.

A seventh aspect combinable with any of the previous aspects furtherincludes one or more nozzles that are fluidly coupled to the pump andpositioned in the vapor condensing layer of the volume.

In another example implementation, a data center cooling system includesa container that defines a volume; a liquid seal to fluidly isolate aliquid phase of a non-conductive coolant that fills at least a portionof the volume from an ambient environment; a plurality of electronicheat-generating devices at least partially immersed in the liquid phaseof the non-conductive coolant to transfer a heat load to thenon-conductive coolant; and a cooling module positioned in the volume tocool the non-conductive coolant.

In a first aspect combinable with the example implementation, thecontainer includes a base and a chimney that extends vertically from thebase, the base including a first portion of the volume and the chimneyincludes a second portion of the volume that is in fluid communicationwith the first portion.

In a second aspect combinable with any of the previous aspects, thesecond portion defines a coolant recovery layer that includes a mixtureof air and a vapor phase of the non-conductive coolant, a vaporcondensing layer that includes substantially the vapor phase of thenon-conductive coolant, and a liquid sub-cooling layer that includessubstantially the liquid phase of the non-conductive coolant.

In a third aspect combinable with any of the previous aspects, thecooling module is positioned in the volume to sub-cool the liquid phaseof the non-conductive coolant in the liquid sub-cooling layer, andcondense the vapor phase of the non-conductive coolant in the vaporcondensing layer.

In a fourth aspect combinable with any of the previous aspects, thecooling module includes a cooling coil that includes a cooling liquidinlet and a cooling liquid outlet, the inlet and the outlet sealinglyextending through the container.

In another example implementation, a server assembly includes a serverboard including a plurality of connector slots coupled to a frontsurface of the server board; a plurality of computing devices installedin at least a portion of the plurality of connector slots to define aserver board topography; and a filler member coupled to the serverboard.

In a first aspect combinable with the example implementation, the fillermember includes a filler surface that faces the front surface of theserver board.

In a second aspect combinable with any of the previous aspects, thefiller member substantially mirrors the server board topography todefine a gap between the server board with the plurality of computingdevices and the filler member.

A third aspect combinable with any of the previous aspects furtherincludes a backing plate mounted to a back surface of the server boardopposite the front surface.

A fourth aspect combinable with any of the previous aspects furtherincludes a filler sub-member installed in a particular one of theplurality of connector slots such that the server board topography isdefined, at least in part, by the installed filler sub-member.

In a fifth aspect combinable with any of the previous aspects, the gapis about 1-2 mm.

In a sixth aspect combinable with any of the previous aspects, the gapis based, at least in part, on a rate of heat transfer from theplurality of computing devices to a cooling fluid enclosed in the gapand in thermal conductive and/or convective contact with the pluralityof computing devices.

In a seventh aspect combinable with any of the previous aspects, thecooling fluid includes a non-conductive liquid.

In an eighth aspect combinable with any of the previous aspects, the gapincludes a boiling zone for the non-conductive liquid.

In a ninth aspect combinable with any of the previous aspects, thefiller member includes a molded member made of a non-porous material.

Various implementations of a data center cooling system according to thepresent disclosure may include one, some, or all of the followingfeatures. For example, the data center cooling system may utilize adielectric, or non-conductive, liquid coolant to cool one or moreelectronic heat-generating devices, such as processors, memory modules(e.g., DIMMs or other memory), networking devices, or otherwise. Thedielectric, or non-conductive, liquid coolant is a liquid coolant that,in some aspects, retards or prevents electric charges from flowingtherethrough, thereby allowing normal operation of the electronicheat-generating devices while immersed in the liquid coolant. As afurther example, the data center cooling system may cool more denselypacked or positioned electronic heat-generating devices as compared toconventional cooling systems. As another example, the data centercooling system may cool higher power electronic heat-generating deviceswithin a similar space as compared to conventional cooling systems.Further, the data center cooling system may, as compared to conventionaldielectric liquid cooling systems, use less dielectric liquid, with alower cost, to cool the electronic devices. The data center coolingsystem may also prevent or substantially prevent the escape of liquid orvapor dielectric coolant. As a further example, the data center coolingsystem may utilize a single phase dielectric, or non-conductive, coolantliquid, thereby substantially preventing vapor bleed-off from theliquid. As yet another example, the data center cooling system mayreduce a volume of space between and among the heat-generating devicesthrough a filler, such as a molded filler. In another example, the datacenter cooling system may provide an extended volume to trap a vaporphase of the dielectric liquid coolant and return the vapor phase to aliquid phase. Thus, as compared to conventional liquid coolant systems,implementations described in the present disclosure may use or requiresubstantially less dielectric liquid coolant.

The details of one or more embodiments are set forth in the accompanyingdrawings and the description below. Other features, objects, andadvantages will be apparent from the description and drawings, and fromthe claims.

DESCRIPTION OF DRAWINGS

FIGS. 1A-1D illustrate various schematic views of an exampleimplementation of a data center cooling system that uses anon-conductive liquid coolant to cool one or more electronicheat-generating devices;

FIG. 2 illustrates a schematic side view of another exampleimplementation of a data center cooling system that uses anon-conductive liquid coolant to cool one or more electronicheat-generating devices;

FIG. 3 illustrates a schematic side view of another exampleimplementation of a data center cooling system that uses anon-conductive liquid coolant to cool one or more electronicheat-generating devices;

FIG. 4A illustrates an example implementation of a server sub-assemblythat may be used in a data center cooling system that uses anon-conductive liquid coolant to cool one or more electronicheat-generating devices;

FIG. 4B-4G illustrate example implementations of a server assembly thatincludes a filler member for use in a data center cooling system thatuses a non-conductive liquid coolant to cool one or more electronicheat-generating devices;

FIG. 5 is a schematic side view of another example implementation of adata center cooling system that uses a non-conductive liquid coolant tocool one or more electronic heat-generating devices; and

FIGS. 6-7 are flowcharts that illustrate example methods of coolingheat-generating devices in a data center with a non-conductive liquidcoolant.

DETAILED DESCRIPTION

This document discusses implementations of a data center cooling systemthat uses a non-conductive, or dielectric, coolant to remove heatgenerated by one or more heat-generating computing devices. The exampledata center cooling systems may fluidly isolate the non-conductivecoolant from an ambient environment in one or multiple containers. Theexample data center cooling systems may cool the non-conductive coolantwith one or more cooling modules contained in the one or multiplecontainers. In some aspects, the computing devices may be mounted orconnected to a structure (e.g., server tray, server board, motherboard,or otherwise) to which a molded filler is coupled or inserted. A side ofthe molded filler may substantially mirror a topography of the structurethat includes the computing devices.

FIGS. 1A-1D illustrate various schematic views of an exampleimplementation of a data center cooling system 100 that uses anon-conductive liquid coolant to cool one or more electronicheat-generating devices. FIG. 1B illustrates a sectional side view ofthe data center cooling system 100. FIG. 1C illustrates a sectionalfront view of the data center cooling system 100. FIG. 1D illustrates asectional top view of the data center cooling system 100. Generally, thesystem 100 includes an outer container 102 that seals the non-conductivecoolant (e.g., liquid and vapor) within the outer container 102. In theillustrated implementation, the outer container 102 comprises a basehousing 106 and a chimney housing (or chimney) 104 that extendsvertically from the base housing 106. In some implementations, thecooling system 100 can be approximately 50 inches wide (e.g., across thefront of the base housing 106), 30 inches deep, and 72 inches tall(e.g., 36 inch height of base housing 106 plus 36 inch extended heightof chimney 104 above the base housing 106).

In the illustrated implementation, access to a volume of the outercontainer 102 is facilitated by a removable cover 108 that includes orcreate a fluid seal between the volume and an ambient environmentexternal to the outer container 102. The cover 108 may provide access toone or more server assemblies 134 (described below) as well as a liquidphase 150 of a non-conductive coolant, as well as other components ofthe system 100. The cover 108 may, in some implementations,substantially prevent any or all liquid or vapor non-conductive coolantfrom exiting the outer container through the base housing 106.

As shown, a relief device 110 may be positioned on the outer container102, such as at a high point of the container 102 on top of the chimney104. The relief device 110 may be a vent, orifice, pressure reliefvalve, or otherwise that allows a flow of air from the volume of theouter container 102 to the ambient environment external to the container102. For example, as a pressure relief valve, the device may be presetto open at a particular pressure (e.g., internal to the container 102)to vent a build-up of air in the container 102. The air may be vented,for example, so that thermodynamic properties or processes within thecontainer (e.g., cooling, condensing, or otherwise) are notsubstantially altered from a desired design. In some aspects, asdescribed below, only air or substantially air, rather than a mix of airand a vapor phase 148 of the non-conductive coolant, may be vented tothe ambient environment.

As illustrated, a cooling liquid supply 112 and a cooling liquid return114 may be fluidly coupled to the system 100, e.g., through the outercontainer 102 and to a cooling module 112 mounted within the volume ofthe container 102. The cooling liquid supply 112 may be, for example, achilled water supply, chilled glycol/refrigerant supply, anevaporatively-cooled liquid, or otherwise (e.g., a liquid coolant thatis cooled through mechanical refrigeration, evaporation, or otherwise).

With references to FIGS. 1B-1D, one or more inner containers 124 aremounted within the volume of the outer container 102. Each of theillustrated inner containers 124 seal the liquid phase 150 and the vaporphase 148 of the non-conductive coolant within a volume of the innercontainer 124 that is in fluid communication, as shown, with the chimney104. A top portion of the inner container 124 may be formed by a cover126 that includes, in this implementation, a handle 128. As discussedlater with reference to FIG. 4A, the cover 126 may form part of thesever assembly 134. In alternative embodiments, the cover 126 may beseparated from the server assembly 134. In any event, the outercontainer 102 and inner container 124 (or containers 124) may form acontainer-in-container system that substantially seals the liquid phase150 and the vapor phase 148 of the non-conductive coolant within theinner container 124 and chimney 104.

The illustrated server assembly 134, as shown, may be verticallypositioned within the inner container 124 and, in this implementation,immersed within the liquid phase 150 of the non-conductive coolant in asub-cooled liquid layer 122. The server assembly 134, in this example,includes one or more memory modules 138 (e.g., DIMMs or other memorymodules), one or more processors 136 (e.g., CPUs or otherwise), and apower interconnect 142. In this example, these components may be mountedon a server board which is mounted to a backing plate 144.

The server assembly also includes one or more I/O patch panels 130 thatare mounted above or to the cover 126 and connected to the memorymodules 138 and/or processors 136 through connectors 132. As shown, theI/O patch panels 130 are positioned above the vapor phase 148 of thenon-conductive coolant within a vapor condensing layer 120 and in anair/vapor mixture 146. The air/vapor mixture 146 may include a mix ofair and the vapor phase 146 of the non-conductive coolant. In someembodiments, the mixture 146 may be substantially or mostly (or all)air.

Further, in this example, and as discussed in more detail below, theillustrated server assembly 134 includes a filler 140 that is mounted toor with the server assembly 134. The filler 140, generally, mayeliminate or reduce an empty volume of the server assembly 134 (e.g., aspace within the volumetric boundaries of the server assembly 134 thatis not taken up by the components of the assembly 134). In some aspects,by reducing the amount of empty volume within the inner container 124,the amount of liquid phase 150 of the non-conductive coolant needed tocool the components of the server assembly 134 (e.g., the memory modules138, processors 136, or otherwise) may also be reduced to save costs.

As illustrated in this implementation, a cooling module 116 is mountedwithin the chimney 104. Although the cooling module 116 shown here is acooling coil (e.g., fin-and-tube heat exchanger), other forms of coolingmodules, such as thermoelectric coolers, Peltier coolers, or otherwise,also are within the scope of the present disclosure. In this example,the cooling module 116 extends through all or most of the height of thechimney 104 (and can also extend a width of the chimney 104 as well). Asillustrated, the cooling module 116 extends through severalthermodynamic layers within the chimney 104 and the volume of the outercontainer 102 generally. At the bottom of the chimney 104, the coolingmodule 116 is positioned in the sub-cooled liquid layer 122, whichcontains all or mostly the liquid phase 150 of the non-conductivecoolant. Here, the cooling module 116 can sub-cool the liquid phase 150,in which the server assemblies 134 are immersed to cool the componentsof the assemblies 134. Towards the middle of the chimney 104, thecooling module 116 extends through the vapor condensing layer 120, whichcontains all or mostly the vapor phase 148 of the non-conductivecoolant. Here, the cooling module 116 cools, and thereby condenses, thevapor phase 146 to the liquid phase 150. Towards the top of the chimney104, the cooling module 116 extends through a coolant recovery layer118, which contains mostly or all air, but could also contain some ofthe vapor phase 148 of the non-conductive coolant. Here, the coolingmodule 116 cools the air to condense all or most of any remaining vaporphase 148 of the non-conductive coolant. Thus, at or near the pressurerelief device 110, all or mostly air (and not costly non-conductivecoolant) is vented to the ambient environment.

With reference to FIG. 1D specifically, each cover 126 includes one ormore vent holes 152 that fluidly couple the volume of the innercontainers 124 with the volume of the outer container 102 in the basehousing 106. For example, the vent holes 152 allow for gaseouscommunication from the air/vapor mixture 146 in the coolant recoverylayer 118 to exit the volume of the inner containers 124 to the volumeof the outer container 104, including the coolant recovery layer 118 inthe chimney 104.

FIG. 2 illustrates a schematic side view of another exampleimplementation of a data center cooling system 200 that uses anon-conductive liquid coolant. Generally, in this implementation, thedata cooling system 200 includes a pump 252 that can circulate a liquidphase 250 of the non-conductive coolant upward from a bottom portion ofan outer container 202 towards an upper portion of a base housing 206 ofthe outer container 202 to cool one or more heat-generating devicesmounted on one or more server assemblies 234. In this implementation, asub-cooled liquid layer 222 may be substantially thinner as compared tothe implementation of the data center cooling system 100, thereby savingcostly non-conductive coolant.

As illustrated in this side view, the system 200 includes an outercontainer 202 that seals the non-conductive coolant (e.g., liquid andvapor) within the outer container 202. In the illustratedimplementation, the outer container 202 comprises a base housing 206 anda chimney housing (or chimney) 204 that extends vertically from the basehousing 206. In some implementations, the cooling system 200 can beapproximately 50 inches wide (e.g., across the front of the base housing206), 30 inches deep, and 72 inches tall (e.g., 36 inch height of basehousing 206 plus 36 inch extended height of chimney 204 above the basehousing 206).

In the illustrated implementation, access to a volume of the outercontainer 202 is facilitated by a removable cover 208 that includes orcreate a fluid seal between the volume and an ambient environmentexternal to the outer container 202. The cover 208 may provide access toone or more server assemblies 234 (described below) as well as a liquidphase 250 of a non-conductive coolant, as well as other components ofthe system 200. The cover 208 may, in some implementations,substantially prevent any or all liquid or vapor non-conductive coolantfrom exiting the outer container through the base housing 206.

As shown, a relief device 210 may be positioned on the outer container202, such as at a high point of the container 202 on top of the chimney204. The relief device 210 may be a vent, orifice, pressure reliefvalve, or otherwise that allows a flow of air from the volume of theouter container 202 to the ambient environment external to the container202. For example, as a pressure relief valve, the device may be presetto open at a particular pressure (e.g., internal to the container 202)to vent a build-up of air in the container 202. The air may be vented,for example, so that thermodynamic properties or processes within thecontainer (e.g., cooling, condensing, or otherwise) are notsubstantially altered from a desired design. In some aspects, asdescribed below, only air or substantially air, rather than a mix of airand a vapor phase 248 of the non-conductive coolant, may be vented tothe ambient environment.

As illustrated, a cooling liquid supply 212 and a cooling liquid return214 may be fluidly coupled to the system 200, e.g., through the outercontainer 202 and to a cooling module 216 mounted within the volume ofthe container 202. The cooling liquid supply 212 may be, for example, achilled water supply, chilled glycol/refrigerant supply, anevaporatively-cooled liquid, or otherwise. (e.g., a liquid coolant thatis cooled through mechanical refrigeration, evaporation, or otherwise).

One or more inner containers 224 are mounted within the volume of theouter container 202. Each of the illustrated inner containers 224 sealthe liquid phase 250 and the vapor phase 248 of the non-conductivecoolant within a volume of the inner container 224 that is in fluidcommunication, as shown, with the chimney 204. A top portion of theinner container 224 may be formed by a cover 226 that includes, in thisimplementation, a handle 228. The cover 226 may form part of the severassembly 234. In alternative embodiments, the cover 226 may be separatedfrom the server assembly 234. In any event, the outer container 202 andinner container 224 (or containers 224) may form acontainer-in-container system that substantially seals the liquid phase250 and the vapor phase 248 of the non-conductive coolant within theinner container 224 and chimney 204.

The illustrated server assembly 234, as shown, may be verticallypositioned within the inner container 224 and, in this implementation,immersed within the liquid phase 250 of the non-conductive coolant in asub-cooled liquid layer 222. The server assembly 234, in this example,includes one or more memory modules 238 (e.g., DIMMs or other memorymodules), one or more processors 236 (e.g., CPUs or otherwise), and apower interconnect 242. In this example, these components may be mountedon a server board which is mounted to a backing plate 244.

The server assembly also includes one or more I/O patch panels 230 thatare mounted above or to the cover 226 and connected to the memorymodules 238 and/or processors 236 through connectors 232. As shown, theI/O patch panels 230 are positioned above the vapor phase 248 of thenon-conductive coolant within a vapor condensing layer 220 and in anair/vapor mixture 246. The air/vapor mixture 246 may include a mix ofair and the vapor phase 246 of the non-conductive coolant. In someembodiments, the mixture 246 may be substantially or mostly (or all)air.

As illustrated in this implementation, a cooling module 216 is mountedwithin the chimney 204. Although the cooling module 216 shown here is acooling coil (e.g., fin-and-tube heat exchanger), other forms of coolingmodules, such as thermoelectric coolers, Peltier coolers, or otherwise,also are within the scope of the present disclosure. In this example,the cooling module 216 extends through all or most of the height of thechimney 204 (and can also extend a width of the chimney 204 as well). Asillustrated, the cooling module 216 extends through severalthermodynamic layers within the chimney 204 and the volume of the outercontainer 202 generally. At the bottom of the chimney 204, the coolingmodule 216 is positioned in the sub-cooled liquid layer 222, whichcontains all or mostly the liquid phase 250 of the non-conductivecoolant. Here, the cooling module 216 can sub-cool the liquid phase 250,in which the server assemblies 234 are immersed to cool the componentsof the assemblies 234. As previously described, the sub-cooled liquidlayer 222 may be substantially thinner or shallower that the sub-cooledliquid layer 122, thereby resulting in less needed liquid phase 250 ofthe non-conductive coolant to cool the server assemblies 234. The pump252, as shown, includes an inlet in the sub-cooled liquid layer 222 andan outlet coupled to a header 254, which in turn is coupled to nozzles258 mounted in the vapor condensing layer 220 and adjacent the serverassemblies 234. The pump 252 may circulate (e.g., constantly, variably,periodically, or otherwise) the liquid phase 250 of the non-conductivecoolant to directly cool (e.g., through thermally conductive and/orconvective contact) the server assemblies 234. Upon such thermallyconductive and/or convective contact, the liquid phase 250 may boil offinto the vapor phase 248 of the non-conductive coolant.

Towards the middle of the chimney 204 (and, as shown, much deeper thanvapor, condensing layer 120) the cooling module 216 extends through thevapor condensing layer 220, which contains all or mostly the vapor phase248 of the non-conductive coolant. Here, the cooling module 216 cools,and thereby condenses, the vapor phase 246 to the liquid phase 250.Towards the top of the chimney 204, the cooling module 216 extendsthrough a coolant recovery layer 218, which contains mostly or all air,but could also contain some of the vapor phase 248 of the non-conductivecoolant. Here, the cooling module 216 cools the air to condense all ormost of any remaining vapor phase 248 of the non-conductive coolant.Thus, at or near the pressure relief device 210, all or mostly air (andnot costly non-conductive coolant) is vented to the ambient environment.

Each cover 226 includes one or more vent holes 252 that fluidly couplethe volume of the inner containers 224 with the volume of the outercontainer 202 in the base housing 206. For example, the vent holes 252allow for gaseous communication from the air/vapor mixture 246 in thecoolant recovery layer 218 to exit the volume of the inner containers224 to the volume of the outer container 204, including the coolantrecovery layer 218 in the chimney 204.

FIG. 3 illustrates a schematic side view of another exampleimplementation of a data center cooling system 300 that uses anon-conductive liquid coolant. In this example implementation, the datacenter cooling system 300 may utilize a single-phase (e.g., liquidphase) non-conductive coolant, e.g., a non-conductive coolant that doesnot vaporize to a vapor phase from a liquid phase based on receiving aheat load from one or more electric heat-generating devices on a serverassembly. Some examples of single-phase non-conductive coolant includemineral oil, fluorocarbons, and other non-conductive coolants that havea flash, or boiling, point less than a temperature (e.g., maximum ratedtemperature) of the electronic heat-generating devices.

In this example implementation, the system 300 includes an outercontainer 302 that seals the non-conductive coolant (e.g., liquid)within the outer container 302. In the illustrated implementation, theouter container 302 comprises a substantially rectangular prismaticvolume (e.g., without a chimney). In some implementations, the coolingsystem 300 can be approximately 50 inches wide (e.g., across the frontof the container 302, not shown in FIG. 3), 30 inches deep, and 36inches tall.

In the illustrated implementation, access to a volume of the outercontainer 302 is facilitated by a removable cover 308 that includes orcreate a fluid seal between the volume and an ambient environmentexternal to the outer container 302. The cover 308 may provide access toone or more server assemblies 334 (described below) as well as theliquid non-conductive coolant 348, as well as other components of thesystem 300. The cover 308 may, in some implementations, substantiallyprevent any or all liquid or vapor non-conductive coolant from exitingthe outer container through the container 302.

As illustrated, a cooling liquid supply 312 and a cooling liquid return314 may be fluidly coupled to the system 300, e.g., through the outercontainer 302 and to a cooling module 316 mounted within the volume ofthe container 302. The cooling liquid supply 312 may be, for example, achilled water supply, chilled glycol/refrigerant supply, anevaporatively-cooled liquid, or otherwise. (e.g., a liquid coolant thatis cooled through mechanical refrigeration, evaporation, or otherwise).

One or more inner containers 324 are mounted within the volume of theouter container 302. Each of the illustrated inner containers 324 sealthe liquid non-conductive coolant 348 within a volume of the innercontainer 324. A top portion of the inner container 324 may be formed bya cover 326 that includes, in this implementation, a handle 328. Asdiscussed later with reference to FIG. 4A, the cover 326 may form partof the sever assembly 334. In alternative embodiments, the cover 326 maybe separated from the server assembly 334. In any event, the outercontainer 302 and inner container 324 (or containers 324) may form acontainer-in-container system that substantially seals the liquidnon-conductive coolant 348 within the outer container 302.

The illustrated server assembly 334, as shown, may be verticallypositioned within the inner container 324 and, in this implementation,immersed within the liquid non-conductive coolant 348 in a liquid layer322. The server assembly 334, in this example, includes one or morememory modules 338 (e.g., DIMMs or other memory modules), one or moreprocessors 336 (e.g., CPUs or otherwise), and a power interconnect 342.In this example, these components may be mounted on a server board whichis mounted to a backing plate 344.

The server assembly also includes one or more I/O patch panels 330 thatare mounted above or to the cover 326 and connected to the memorymodules 338 and/or processors 336 through connectors 332. As shown, theI/O patch panels 330 are positioned above the liquid non-conductivecoolant 348 and within an air layer 320 that contains mostly or onlyair.

In the illustrated implementation, the server assemblies 334 do notinclude a filler as described above with respect to the serverassemblies 134. Although not shown, the illustrated server assembly 334can include a filler that is mounted to or with the server assembly 334.The filler, generally, may eliminate or reduce an empty volume of theserver assembly 334 (e.g., a space within the volumetric boundaries ofthe server assembly 334 that is not taken up by the components of theassembly 334). In some aspects, by reducing the amount of empty volumewithin the inner container 324, the amount of liquid non-conductivecoolant 348 needed to cool the components of the server assembly 334(e.g., the memory modules 338, processors 336, or otherwise) may also bereduced.

As illustrated in this implementation, a cooling module 316 is mountedwithin the volume of the container 302. Although the cooling module 316shown here is a cooling coil (e.g., fin-and-tube heat exchanger), otherforms of cooling modules, such as thermoelectric coolers, Peltiercoolers, or otherwise, also are within the scope of the presentdisclosure. In this example, the cooling module 316 extends through allor most of the height of the container 302 (and can also extend a widthof the outer container 302 as well). As illustrated, the cooling module316 extends through several thermodynamic layers within the outercontainer 302 and the volume of the outer container 302, generally. Atthe bottom of the outer container 302, the cooling module 316 ispositioned in the liquid layer 322, which contains all or mostly theliquid non-conductive coolant 348. Here, the cooling module 316 can coolor sub-cool the liquid non-conductive coolant 348, in which the serverassemblies 334 are immersed, to cool the components of the assemblies334. At or near a top of the outer container 302, the cooling module 316can extend through the air layer 320, but the heat exchanger portion ofthe cooling module 316 may reside all or mostly in the liquid layer 322.

A pump 352, as shown, includes an inlet in a bottom portion of theliquid layer 222 and an outlet coupled to a header 354, which in turnincludes an outlet near or adjacent a top end of the cooling module 316.The pump 352 may circulate (e.g., constantly, variably, periodically, orotherwise) the liquid non-conductive coolant 348 within the volume ofthe outer container 302 (e.g., from bottom to top) to, e.g., ensure evencooling of the liquid non-conductive coolant 348 by the cooling module316 (so as to remove the heat transferred from the server assemblies 324to the liquid non-conductive coolant 348), and ensure cooling of theserver assemblies 334 through thermally conductive and/or convectivecontact. Upon such thermally conductive and/or convective contact, theliquid non-conductive coolant 348 may cool the heat-generatingcomponents without boiling or vaporizing.

In some example implementations, a pressure relief device (not shown)may be mounted to a top of the outer container 302. The relief devicemay be a vent, orifice, pressure relief valve, or otherwise that allowsa flow of air from the volume of the outer container 302 to the ambientenvironment external to the container 302. For example, as a pressurerelief valve, the device may be preset to open at a particular pressure(e.g., internal to the container 302) to vent a build-up of air in thecontainer 302. The air may be vented, for example, so that thermodynamicproperties or processes within the container (e.g., cooling orotherwise) are not substantially altered from a desired design.

Each cover 326 may also include one or more vent holes (not shown) thatfluidly couple the volume of the inner containers 324 with the volume ofthe outer container 302. For example, the vent holes allow for gaseouscommunication from the air layer 320 to exit the volume of the innercontainers 324 to the volume of the outer container 304.

FIG. 4A illustrates an example implementation of a server assembly 400that may be used in a data center cooling system that uses anon-conductive coolant to cool one or more electronic heat-generatingdevices. As illustrated, the examine server assembly 400 includes aserver board 408 mounted to a backing plate 406, both of which aremounted to or with a cover 402 that includes a handle 404. The cover 402can be used, as described above, as a seal or part of a seal for acontainer (e.g., an inside container) of a data center cooling systemthat uses a non-conductive fluid coolant (e.g., single-phase ortwo-phase fluid). As described above, one or more electronicheat-generating components, such as server devices, processors, memorymodules, networking devices, can be mounted to the server board 408. Asa seal, or by including a seal (e.g., an O-ring or other form of seal),the cover 402 may fluidly separate liquids/vapors enclosed in onecontainer from another container or the ambient environment.

FIG. 4B-4G illustrates example implementations of server sub-assembliesthat include a filler member for use in a data center cooling systemthat uses a non-conductive liquid coolant. FIG. 4B shows a side view ofa server sub-assembly 420 that, in some aspects may be a portion of orused with the server assembly 400, and includes a filler 430. Serversub-assembly 420 includes a server board 422 to which one or moreprocessors 424 are mounted and one or more memory modules 428 areconnected through slots 426. As illustrated, the server sub-assembly 420defines a volume 427 into which the processors 424 and memory modules428 extend to fill a portion of the volume 427. The filler 430, may beformed (e.g., molded or otherwise) to fill most, if not all, of the restof the volume 427 not filled by the processors 424 and memory modules428. For example, the filler 430 may be formed so that, when coupled tothe server sub-assembly 420 or within the volume 427, a gap 429 is leftbetween the server board 422, processors 424, and memory modules 428,and the filler 430. This gap 429 may provide a small volume (e.g.,relative to the volume 427) through which a liquid phase of thenon-conductive coolant may flow (or reside), and into which the liquidphase may boil off into a vapor phase. By reducing a volume taken up bythe non-conductive coolant from the volume 427 to the gap 429, lessnon-conductive coolant may be necessary to cool the server sub-assembly420.

FIG. 4C shows a side view of a server sub-assembly 440 that, in someaspects may be a portion of or used with the server assembly 400, andincludes a filler 450 and filler members 452. Server sub-assembly 440includes a server board 442 to which one or more processors 444 aremounted and one or more memory modules 448 are connected through slots446. As illustrated, the server sub-assembly 440 defines a volume 447into which the processors 444 and memory modules 448 extend to fill aportion of the volume 447. The filler 450, may be formed (e.g., moldedor otherwise) to fill most, if not all, of the rest of the volume 447not filled by the processors 444 and memory modules 448. For example,the filler 450 may be formed so that, when coupled to the serversub-assembly 440 or within the volume 447, a gap 449 is left between theserver board 442, processors 444, and memory modules 448, and the filler450.

In this example, one or more slots 446 may not have a correspondingmemory module 448 mounted therein. In such a case, the filler member 452may be formed (e.g., molded) into a similar shape as the absent memorymodule 448 and mounted within the empty slot 446. This gap 449, whichmay provide a small volume (e.g., relative to the volume 447) throughwhich a liquid phase of the non-conductive coolant may flow (or reside),and into which the liquid phase may boil off into a vapor phase, maythus be relatively uniform around the heat-generating components evenwhen some of the components are missing. By reducing a volume taken upby the non-conductive coolant from the volume 447 to the gap 449, lessnon-conductive coolant may be necessary to cool the server sub-assembly440. Further, if empty slots 446 (or other portions of the serversub-assembly 440) are later needed for memory modules 448, the fillermembers 452 may be removed. Thus, in this example, the filler 450 andfiller members 452 (which may be customizable to fit into any volumeshape of the server sub-assembly 440) may provide for an adjustable andchangeable filler so that an amount of non-conductive coolant needed tocool the server sub-assembly may be reduced.

FIGS. 4D-4E illustrate cross-section side views of another exampleimplementation of a server assembly 460. Server assembly 460 includes afiller 467 that, as described above, can be mounted or coupled betweentwo face-to-face server boards 463 to reduce an amount of total volumetaken within a data center cooling system that uses a non-conductivecoolant as described herein. Server assembly 460 includes a cover 461with a handle 468. In some aspects, the cover 461 can be used, asdescribed above, as a seal or part of a seal for a container (e.g., aninside container) of a data center cooling system that uses anon-conductive fluid coolant (e.g., single-phase or two-phase fluid).Backing plates 462 are coupled to or mounted with the cover 461 andsupport server boards 463, which are mounted such that one or more heatgenerating computing devices on each board 463 are facing. The computingdevices can include, as shown, memory modules 464 and processors 465 (aswell as a variety of other computing devices that generate heat). Asshown, a filler 467, such as the fillers 430 or 450, may be mountedwithin the volume between the boards 463, memory modules 464, andprocessors 465. In this example implementation, further space for serverassemblies 460 may be saved in a data center cooling system, such as thesystems 100, 200, 300, or otherwise, so that less non-conductive coolantmay be required and/or the server assemblies 460 may be more denselypacked within the system.

FIG. 4F illustrates a cross-section side view of another exampleimplementation of a server assembly 470. Server assembly 470 includes afiller 477 that, as described above, can be mounted or coupled betweentwo face-to-face server boards 473 to reduce an amount of total volumetaken within a data center cooling system that uses a non-conductivecoolant as described herein. Backing plates 472 are coupled to ormounted with a cover and support server boards 473, which are mountedsuch that one or more heat generating computing devices on each board473 are facing. The computing devices can include, as shown, memorymodules 474 and processors 475 (as well as a variety of other computingdevices that generate heat). As shown, a filler 477, such as the fillers430 or 450, may be mounted within the volume between the boards 473,memory modules 474, and processors 475. In this example implementation,further space is saved by staggering the memory modules 474 from theserver boards 473. For instance, as shown, the memory modules 474 areinterleaved so that the server boards 473 can be more closely positionedwhile facing each other. Thus, further space for server assemblies 470may be saved in a data center cooling system, such as the systems 100,200, 300, or otherwise, so that less non-conductive coolant may berequired and/or the server assemblies 470 may be more densely packedwithin the system.

FIG. 4G illustrates a cross-section side view of another exampleimplementation of a server assembly 490. Server assembly 490 includes afiller 497 that, as described above, can be mounted or coupled betweentwo face-to-face server boards 493 to reduce an amount of total volumetaken within a data center cooling system that uses a non-conductivecoolant as described herein. In this example, there are no backingplates, because the filler 497 provides structure support for the serverboards 493, which are mounted such that one or more heat generatingcomputing devices on each board 493 are facing. The computing devicescan include, as shown, memory modules 494 and processors 495 (as well asa variety of other computing devices that generate heat). As shown, afiller 497, such as the fillers 430 or 450, may be mounted within thevolume between the boards 493, memory modules 494, and processors 495.As with server assembly 470, further space is saved by staggering thememory modules 494 from the server boards 493. For instance, as shown,the memory modules 494 are interleaved so that the server boards 493 canbe more closely positioned while facing each other. Thus, by eliminatingthe backing plates and interleaving the memory modules 494, furtherspace for server assemblies 490 may be saved in a data center coolingsystem, such as the systems 100, 200, 300, or otherwise, so that lessnon-conductive coolant may be required and/or the server assemblies 490may be more densely packed within the system.

FIG. 5 is a schematic side view of another example implementation of adata center cooling system 500 that uses a non-conductive coolant tocool one or more electronic heat-generating devices. Generally, the datacenter cooling system 500 includes a container-in-container approach tocooling the one or more electronic heat-generating devices with anon-conductive coolant, in which an outer container is ahuman-occupiable structure.

As shown, the data center cooling system 500 includes a number ofcontainers 504 positioned within a human-occupiable structure 502. Eachcontainer 504 may include a vent 522 and be fluidly connected to acooling fluid supply 506 and a cooling fluid return 508. Generally, eachof the containers 504 may contain one or more server assemblies (asdescribed above) that include one or more electronic heat-generatingdevices immersed in a non-conductive coolant (e.g., a liquid or vaporphase of the non-conductive coolant), and may also include one or morecooling modules coupled to the cooling fluid supply 506 and a coolingfluid return 508. Each of the containers 504 may fluidly isolate thenon-conductive coolant (e.g., whether in liquid or vapor form) from ahuman-occupiable workspace defined within the human-occupiable structure502. The vent 522 may allow an amount of air to escape the container504, e.g., based on a pressure within the container 504. In someaspects, the vent 522 may allow a mixture of air and a vapor phase ofthe non-conductive coolant to escape into the human-occupiable structure502.

The illustrated data center cooling system 500 includes one or morecooling modules 512. The cooling module 512 is shown in a plenum orattic space, but the module 512 may be positioned in other locations,such as within the human-occupiable workspace, on a roof of thehuman-occupiable structure 502, or otherwise. Generally, the coolingmodule 512 has an inlet in airflow communication with a return airflow510 from the human-occupiable workspace so that the return airflow 510can be circulated (e.g., by a fan in the cooling module 512) through acooling coil (e.g., in the cooling module 512). In some aspects, thecooling module 512 may receive the return airflow 510, cool the returnairflow 510 to provide a supply airflow 518 to the human-occupiableworkspace, and condense the vapor-phase of the non-conductive coolant,which can be returned to the containers 504 in a conduit 524.

The illustrated data center cooling system 500 can also include one ormore exhaust modules 514. For example, in some cases, cooling is notneeded for the human occupiable structure 502, but air changes may berequired by code or otherwise. In such example cases, the exhaust module514 may circulate room airflow 517 with a fan (e.g., mounted in theexhaust module 514), clean the airflow 517 through a filter (e.g., inthe exhaust module 514) if necessary, and circulate an exhaust airflow520 to an ambient environment. In some aspects, heating or coolingdevices may be part of or used with the exhaust module 514. In someaspects, both cooling modules 512 and exhaust modules 514 may be used inthe data center cooling system 500. In some aspects, only coolingmodules 512 or exhaust modules 514, but not both, are used in the datacenter cooling system 500.

The illustrated human-occupiable structure 502, as mentioned above, mayprovide for or be an outer container in a container-in-container systemthat uses a non-conductive coolant to cool server assemblies, while thecontainers 504 are the inner containers. In some aspects, the liquidphase of the non-conductive coolant is contained in the containers 504(and further contained by the structure 502) so as to minimize orprevent exposure of the non-conductive coolant to the ambientenvironment. Further, the vapor phase of the non-conductive coolant (ifapplicable), may be largely contained in the containers 504. And asdescribed above, the vapor phase of the non-conductive coolant may becondensed and returned to the containers 504. In some aspects, the vaporphase of the non-conductive coolant, as well as air, may be vented tothe ambient environment from the human-occupiable structure 502 througha vent 516.

FIGS. 6-7 are flowcharts that illustrate example methods of coolingheat-generating devices in a data center with a non-conductive coolant.FIG. 6 shows method 600, which may begin by enclosing electronicheat-generating devices (e.g., processors, memories, networking devices,and otherwise) in a volume defined by a first container in step 602. Insome aspects, the devices are mounted on a server board of a serverassembly, such as the server assembly 400.

Method 600 may continue by immersing the heat-generating devices in aliquid phase of a non-conductive coolant in step 604. The non-conductivecoolant may be one of several multi-phase non-conductive, or dielectriccoolants, such as aromatics, silicate-esters, aliphatics, or silicones,as well as single-phase coolants such as mineral oil or fluorocarbons.The non-conductive coolant, generally, may allow immersion of theelectronic devices without conducting and electric charge, or conductingsuch a small electric charge that operation of the electronic devices isnot affected.

Method 600 may continue by enclosing the first container in a secondvolume of a second container in step 606. For example, the firstcontainer may be smaller such that many of the first containersenclosing the heat-generating devices may fit in the volume of thesecond container. In some aspects, the second container is ahuman-occupiable structure. In some implementations, the first containeris formed, at least in part by a portion of a server assembly, such as acover that sealingly engages with the first container. In some aspects,the second container is a two-part container (e.g., attached orintegrally formed), which includes a base section and chimney sectionthat extends above a top of the base section.

Method 600 may continue by sealing the liquid phase of thenon-conductive coolant from an ambient environment in step 608. Forexample, in some aspects, the liquid phase, and in some aspects a vaporphase (e.g., for multi-phase coolants) of the non-conductive coolant maybe sealed within the second container from the ambient environment, butable to flow within the first and second containers to cool theelectronic devices. In some aspects, portions of both the first andsecond containers are flooded with the liquid phase of thenon-conductive coolant, while also enclosing the vapor phase of thenon-conductive coolant. In some aspects, such as for single-phasecoolants, only a liquid phase of the non-conductive coolant fillsportions of the first and second containers. In some aspects, such aswhen the second container is a human-occupiable structure, the liquidphase of the non-conductive coolant may be contained completely orsubstantially within the first container and the vapor phase of thenon-conductive coolant may be mostly contained in the first containerbut could be vented to the second container as well.

Method 600 may continue by transferring a heat load from theheat-generating devices to the liquid phase of the non-conductivecoolant in step 610. For example, as the liquid phase of thenon-conductive coolant comes into thermally conductive and convectivecontact with the electronic devices, the heat load from the devicesboils off all or a portion of the liquid phase in contact with thedevices. The vaporization of the liquid phase removes the heat from theelectronic devices, thereby maintaining a particular or desiredtemperature of the devices during operation.

Method 600 may continue by forming an airflow path between the first andsecond volumes in step 612. For example, in some aspects, such as when atwo-phase non-conductive coolant is used to cool the electronic devices,air or a mixture of air and a vapor phase of the non-conductive coolantmay be vented from the first container to the second container, e.g., tomaintain a particular pressure in the first container or otherwise. Insome aspects, the airflow path may be formed by one or more holes in aportion of a server assembly, such as a cover, which forms a portion ofthe first container.

Method 600 may continue by venting a portion of air from the firstvolume, through the second volume, into the ambient environment in step614. In some aspects, air that accumulated within the second container,such as air that has been vented into the second container from thefirst container, is vented to the ambient environment. In some aspect,the air is a mixture of air and the vapor phase of the non-conductivecoolant. In some aspects, the venting may occur to maintain a particularpressure, such as within the second container. The venting may becontrolled, e.g., with a pressure relief device that vents the portionof air based, at least in part, on a pressure of the second container.

Method 600 may continue by cooling the non-conductive coolant in step616. In some aspects, this heat load that is transferred to thenon-conductive coolant is further transferred from the non-conductivecoolant to another cooling medium, such as a cooling liquid circulatedto the first container to remove the heat from the non-conductivecoolant. In some aspects, a cooling module may be mounted within thesecond container (or in some aspects, the first container), to receivethe cooling liquid to cool the non-conductive coolant. In some aspects,a flow of cooling liquid is not circulated to the first or secondcontainers, and the cooling module comprises a thermoelectric orPeltier-type cooling module. In some aspects, the liquid phase of thenon-conductive coolant is circulated to flow or spray over theelectronic devices and/or the cooling module.

FIG. 7 shows method 700, which may be or include a more detailedimplementation of step 616 of method 600. For example, method 700 maybegin by cooling a mix of air and a portion of a vapor phase of thenon-conductive coolant in step 702. In some aspects, this cooling may beperformed by a cooling module (e.g., a cooling coil) enclosed with thenon-conductive coolant. The cooling module may extend through one ormore thermodynamic layers of one or more containers of the data centercooling system. In some aspects, the mix of air and the portion of thevapor phase of the non-conductive coolant may be at or near a topportion of one or more of the containers in the cooling system based on,for instance a relative density of the mixture (e.g., relative to otherfluids in the containers).

Method 700 may continue by condensing the portion of the vapor phase toa liquid phase of the non-conductive coolant in step 704. As thenon-conductive coolant can be a costly expense in data center coolingsystems, capturing and retaining as much of it as possible, rather thanventing the vapor phase of the non-conductive coolant (e.g., along withair), may be preferable. Thus, in some aspects, the cooling module maybe designed (e.g., by cooling capacity, entering cooling liquidtemperature and/or flow rate, or otherwise), to cool the air and vaporphase mixture at a temperature just below a dew point of thenon-conductive coolant. In some aspects, the dew point of thenon-conductive coolant may be higher than a dew point of the air in themixture, thereby ensuring that water does not condense from the air.

Method 700 may continue by cooling another portion of the vapor phase ofthe non-conductive coolant in step 706. For example, in some aspects,the fluids in one or more containers in the data center cooling systemmay be substantially stratified. The air and vapor phase mixture may benear a top of the one or more container as described above. Anotherportion of the vapor phase may make up a layer below the air and vaporphase mixture that is primarily vapor phase of the non-conductivecoolant. The cooling module may extend through the primarily vapor phaselayer and cool this layer.

Method 700 may continue by condensing the other portion of the vaporphase to the liquid phase of the non-conductive coolant in step 708. Asin step 704, the vapor phase (e.g., in the primarily vapor phase layer)of the non-conductive coolant may be condensed by cooling such fluidbelow its dew point. Once condensed, the liquid phase of thenon-conductive coolant may naturally circulate (e.g., by density)downward in the one or more containers of the data center coolingsystem.

Method 700 may continue by sub-cooling a liquid phase of thenon-conductive coolant in step 710. For example, in some aspects, abottom layer (e.g., at or near a bottom of the one or more containers ofthe data center cooling system) may be or include a liquid phase of thenon-conductive coolant. The cooling module may sub-cool the liquid phaseof the non-conductive coolant to, for instance, ensure that all orsubstantially all of the non-conductive coolant in which the electronicdevices are immersed is in the liquid phase, rather than a multi-phasenon-conductive coolant of liquid and vapor. In some aspects, by ensuringthat all or substantially all of the non-conductive coolant in which theelectronic devices are immersed is in the liquid phase, heat transferfrom the electronic devices to the non-conductive coolant is increasedand/or optimized.

Method 700 may continue by circulating the liquid phase of thenon-conductive coolant to a vapor phase layer, such as the primarilyvapor phase layer, of the one or more containers in step 712. Forexample, in some aspects, all of the electronic devices (e.g., mountedon a server assembly) may not be immersed in the liquid phase of thenon-conductive coolant. Some of the electronic devices may be positionedwithin the one or more containers at the primarily vapor phase layer.The liquid phase of the non-conductive coolant may be circulated (e.g.,by a pump positioned with an inlet in liquid communication with theliquid phase of the non-conductive coolant) to contact (e.g., by sprayor otherwise) the electronic devices positioned within the primarilyvapor phase layer.

A number of embodiments have been described. Nevertheless, it will beunderstood that various modifications may be made without departing fromthe spirit and scope of what is described. For example, the steps of theexemplary flow charts in FIGS. 6-7 may be performed in other orders,some steps may be removed, and other steps may be added. As anotherexample, data center cooling systems that utilize acontainer-in-container concept as described herein may not use a chimneyas part of an outer container, but instead use a substantiallyrectangular (or square) prismatic volume. Further, some implementationsthat use a single container (e.g., only an outside container) may or maynot utilize a chimney. As another example, a data center cooling systemthat uses a single-phase non-conductive coolant according to the presentdisclosure may use a container-in-container approach or a singlecontainer approach, with or without a chimney as described herein.Accordingly, other embodiments are within the scope of the followingclaims.

What is claimed is:
 1. A data center cooling system, comprising: anouter container that defines a first volume, the outer containercomprising a base and a chimney that extends vertically from the baseand above a portion of a top surface of the outer container, the basecomprising a first portion of the first volume and the chimney comprisesa second portion of the first volume that is in fluid communication withthe first portion; an inner container that defines a second volume andis positioned within the first volume, the inner container comprising anair outlet that comprises an airflow path between the first and secondvolumes; a liquid seal to fluidly isolate a liquid phase of anon-conductive coolant that fills at least a portion of the first andsecond volumes from an ambient environment; a plurality of electronicheat-generating devices at least partially immersed in the liquid phaseof the non-conductive coolant to transfer a heat load to thenon-conductive coolant; and a cooling module mounted in the first volumeand at least partially immersed in the liquid phase of thenon-conductive coolant to cool the liquid phase of the non-conductivecoolant, wherein the cooling module is positioned within the base andthe chimney and extends through the first and second portions of thefirst volume.
 2. The data center cooling system of claim 1, wherein theouter container comprises a pressure relief valve configured to vent aportion of air, vented from the second volume through the air outlet andinto the first volume, to the ambient environment.
 3. The data centercooling system of claim 1, wherein the second portion defines a coolantrecovery layer that comprises a mixture of air and a vapor phase of thenon-conductive coolant, a vapor condensing layer that comprisessubstantially the vapor phase of the non-conductive coolant, and aliquid sub-cooling layer that comprises substantially the liquid phaseof the non-conductive coolant.
 4. The data center cooling system ofclaim 1, wherein the cooling module is at least partially immersed in avapor phase of the non-conductive coolant to condense the vapor phase ofthe non-conductive coolant.
 5. The data center cooling system of claim1, wherein the cooling module comprises a cooling coil that comprises acooling fluid inlet and a cooling fluid outlet.
 6. The data centercooling system of claim 1, further comprising: a pump positioned in theliquid sub-cooling layer of the second portion of the first volume; andone or more nozzles that are fluidly coupled to the pump and positionedin the vapor condensing layer of the second portion of the first volume.7. The data center cooling system of claim 6, wherein the one or morenozzles is fluidly coupled to the pump through one or more conduits thatextend from the first volume into the second volume.
 8. The data centercooling system of claim 1, wherein the non-conductive coolant comprisesa single-phase non-conductive coolant, the system further comprising: apump positioned in a liquid sub-cooling layer of the first volume; andone or more nozzles that are fluidly coupled to the pump and positionedin a vapor condensing layer of the first volume, wherein the coolingmodule is at least partially immersed in a sub-cooled portion of thesingle-phase non-conductive coolant in the liquid sub-cooling layer ofthe first volume.
 9. The data center cooling system of claim 1, whereinthe outer container comprises a human-occupiable housing, and the firstvolume comprises a human-occupiable workspace.
 10. The data centercooling system of claim 1, wherein the non-conductive coolant comprisesa dielectric coolant.
 11. A method for cooling electronicheat-generating devices in a data center, comprising: enclosing aplurality of electronic heat-generating devices in a first volumedefined by a first container; immersing the plurality of electronicheat-generating devices in a liquid phase of a non-conductive coolant;enclosing the first container in a second volume of a second container,the non-conductive coolant filling at least a portion of the first andsecond volumes, the second container comprising a base and a chimneythat extends vertically from the base and above a portion of a topsurface of the first container, the base comprising a first portion ofthe second volume and the chimney comprising a second portion of thesecond volume that is in fluid communication with the first portion;sealing the liquid phase of the non-conductive coolant from an ambientenvironment; transferring a heat load from the plurality of electronicheat-generating devices to the liquid phase of the non-conductivecoolant; cooling the liquid phase of the non-conductive coolant with acooling module that is at least partially immersed within the liquidphase of the non-conductive coolant in the first volume of the firstcontainer and extends through the first and second portions of thesecond volume; and cooling, with the cooling module, a mix of air and afirst portion of a vapor phase of the non-conductive coolant in a topportion of the chimney to condense the first portion of the vapor phaseto the liquid phase of the non-conductive coolant.
 12. The method ofclaim 11, further comprising forming an airflow path between the firstand second volumes.
 13. The method of claim 11, further comprisingventing a portion of air from the first volume, through the airflowpath, through the second volume and to the ambient environment.
 14. Themethod of claim 11, further comprising cooling, with the cooling module,a second portion of the vapor phase of the non-conductive coolant in amiddle portion of the chimney to condense the second portion of thevapor phase to the liquid phase of the non-conductive coolant.
 15. Themethod of claim 11, further comprising sub-cooling, with the coolingmodule, the liquid phase of the non-conductive coolant in a bottomportion of the chimney.
 16. The method of claim 15, further comprising:circulating a portion of the sub-cooled liquid phase of thenon-conductive coolant from the bottom portion of the chimney to thefirst volume to contact the plurality of electronic heat-generatingdevices.
 17. The method of claim 11, further comprising supplying acooling fluid to the cooling module positioned in the chimney.
 18. Themethod of claim 11, wherein the non-conductive coolant comprises asingle-phase non-conductive coolant, the method further comprising:circulating a sub-cooled liquid phase of the non-conductive liquidcoolant from a bottom portion of the second volume to a top portion ofthe second volume; and circulating the sub-cooled liquid in the topportion over the cooling module positioned in the second volume.
 19. Themethod of claim 11, wherein the second container comprises ahuman-occupiable housing, and the second volume comprises ahuman-occupiable workspace.
 20. The method of claim 11, wherein thenon-conductive coolant comprises a dielectric coolant.
 21. A systemcomprising: a first housing adapted to enclose a plurality of serverassemblies, at least one of the server sub-assemblies comprising a topthat sealingly engages the first housing and a server board coupled tothe top; a second housing that encloses the first housing, the secondhousing comprising a base housing and an extension housing that iscoupled to a lateral side of the base housing and extends verticallyfrom the base housing; a dielectric coolant enclosed within the firstand second housings to cool the server board, the dielectric coolantcomprising a liquid phase substantially contained within the secondhousing and a vapor phase substantially contained within the firsthousing; and a cooling coil positioned in the extension housing andmounted in the second volume to cool the liquid phase of the dielectriccoolant, and condense the vapor phase of the dielectric coolant in theextension housing to the liquid phase of the dielectric coolant, thecooling coil at least partially immersed in the liquid phase of thedielectric coolant to cool the liquid phase of the dielectric coolant.22. The system of claim 21, further comprising: a plurality of computingdevices mounted on the server board; and an I/O patch panel mounted onthe top and communicably coupled to the plurality of computing devicesmounted on the server board.
 23. The system of claim 21, furthercomprising a pump positioned in the liquid phase of the dielectriccoolant to circulate the liquid phase of the dielectric coolant to avapor phase layer of the first housing.
 24. The data center coolingsystem of claim 3, wherein the cooling module extends from the coolantrecovery layer, through the vapor condensing layer, into the liquidsub-cooling layer to condense the vapor phase of the non-conductivecoolant and cool the liquid phase of the non-conductive coolant.
 25. Thedata center cooling module of claim 1, wherein the portion of the topsurface of the outer container comprises a top outer surface of thebase.
 26. The data center cooling module of claim 25, wherein the liquidseal comprises a removable cover mounted on the top outer surface of thebase.